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Preface 


This  report  documents  the  structured  process  that  was  developed  to  help  project  teams 
identify  acquisition  framing  assumptions.  The  first  chapter  introduces  the  process 
and  describes  the  overall  approach.  The  process  begins  in  a  workshop  facilitated  by  a 
PowerPoint  presentation.  The  slides  from  this  presentation  are  shown  in  Chapter  Two. 
This  research  was  sponsored  by  the  Performance  Assessment  and  Root  Cause  Analysis 
(PARCA)  office  and  conducted  within  the  Acquisition  and  Technology  Policy  Center 
of  the  RAND  National  Defense  Research  Institute,  a  federally  funded  research  and 
development  center  sponsored  by  the  Office  of  the  Secretary  of  Defense,  the  Joint  Staff, 
the  Unified  Combatant  Commands,  the  Navy,  the  Marine  Corps,  the  defense  agen¬ 
cies,  and  the  defense  Intelligence  Community. 

For  more  information  on  the  RAND  Acquisition  and  Technology  Policy  Center 
Center,  see  http://www.rand.org/nsrd/ndri/centers/atp.html  or  contact  the  director 
(contact  information  is  provided  on  the  web  page). 
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CHAPTER  ONE 


Structured  Process  Guidance 


Overview 

Employment  of  acquisition  framing  assumptions  (FAs)  is  an  approach  to  defining  and 
tracking  key  program  assumptions  that  are  made  early  in  program  development  and 
throughout  the  program  life.  FAs  serve  as  a  form  of  risk  analysis  to  identify  uncertain¬ 
ties  that  may  or  may  not  be  recognized  as  such.  This  document  provides  supporting 
information  for  the  briefing  titled  “Identifying  and  Prioritizing  Acquisition  Framing 
Assumptions:  Structured  Deliberation  Exercise” — a  briefing  to  be  used  in  a  group  set¬ 
ting  to  identify  FAs.  It  also  provides  a  brief  introduction  to  the  concept  of  FAs;  an 
overview  of  the  approach  used  in  the  briefing  to  identify  FAs;  and  a  discussion  of  how 
to  operationalize  this  approach,  including  an  overview  of  structured  deliberation,  some 
of  the  important  concepts  in  having  a  successful  session,  and  tailoring  questions  on 
program  risk  areas  to  help  elucidate  FAs.  Note  that  the  briefing  is  meant  as  a  starting 
point  and  should  be  tailored  to  specific  program  circumstances. 


Introduction  to  Framing  Assumptions 

In  2013,  a  RAND  report  included  a  definition  of  acquisition  FAs  along  with  a  series 
of  program  examples.1  This  concept  grew  out  of  root  cause  analysis  work  by  the  Per¬ 
formance  Assessment  and  Root  Cause  Analyses  (PARCA)  office  on  Nunn-McCurdy 
breaches  on  programs.  Many  of  these  breaches  were  associated  with  an  incorrect  or 
failing  foundational  assumptions — FAs.  By  making  such  FAs  more  explicit  early  in  the 
program  life  cycle  and  tracking  them,  the  Office  of  the  Secretary  of  Defense  (OSD), 
and  the  Services  may  be  able  to  better  manage  major  risks  to  and  expectations  of 
programs. 

An  FA  is  any  explicit  or  implicit  assumption  that  is  central  in  shaping  cost,  sched¬ 
ule,  or  performance  expectations.  FAs  may  change  over  the  course  of  execution  or  new 


1  Arena,  Doll,  and  McKernan,  2013. 
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ones  can  be  added.  The  PARCA  office  updated  the  criteria  for  FAs2  as  follows  (italics 
in  original): 

•  “ Critical :  Significantly  affects  program  expectations.  This  criterion  means  that  FAs, 
when  they  fail  or  are  incorrect,  will  have  significant  cost,  schedule  and/or  per¬ 
formance  effects  on  the  program.  One  possible  consequence  is  a  formal  program 
breach.  The  criterion  is  meant  to  exclude  the  many  smaller  assumptions  made  for 
a  program  that  do  not  result  in  significant  consequences. 

•  No  workarounds:  Consequences  cannot  be  easily  mitigated.  This  criterion  implies 
that  valid  FAs  have  no  workarounds  or  potential  fixes  if  they  are  wrong.  The  con¬ 
sequences  of  a  failed  FA  will  occur.  When  the  FA  is  wrong,  there  will  be  signifi¬ 
cant  cost  and/or  schedule  implications. 

•  Foundational:  Not  derivative  of  other  assumptions.  This  criterion  is,  perhaps,  the 
hardest  to  understand  and  define.  An  FA  is  foundational  if  it  is  a  high-level  and 
encompassing  assumption.  An  FA  might  have  derivative  assumptions  associated 
with  it,  but  a  proper  FA  will  not  be  derivative  or  subordinate  to  other  major 
assumptions.  The  relationship  between  foundational  and  derivative  assumptions 
can  be  exemplified  by  the  F-35  program.3  There  were  at  least  two  important 
assumptions  related  to  the  program  that  later  turned  out  to  be  incorrect.  The  first 
was  that  a  high  degree  of  concurrency  between  engineering  and  production  was 
acceptable.  The  other  was  that  testing  would  be  more  efficient  than  seen  histori¬ 
cally.  However,  these  are  both  derivative  assumptions  of  the  true  FAs.  The  foun¬ 
dational  FAs  were  that  the  design  technology  was  mature  and  that  the  competi¬ 
tive  prototype  was  production  representative. 

•  Program  specific:  Not  generally  applicable  to  all  programs.  This  criterion 
implies  that  FAs  should  reflect  some  unique  aspects  of  the  program.  For  example, 
an  FA  is  not,  ‘The  contract  will  perform  well.’  However,  an  FA  might  be,  ‘The 
key  technologies  are  sufficiently  mature  such  that  no  component  development  or 
prototyping  is  necessary.’” 

In  the  root  cause  research,  we  observed  that  FAs  can  be  grouped  into  four  major 
areas,  shown  in  Table  1.1. 

Several  examples  were  identified  of  specific  FAs  that  failed  and  contributed  to 
Nunn-McCurdy  breaches: 

•  Joint  Strike  Fighter  (F-35) 

-  Competitive  prototypes  are  production  representative  (discussed  above). 


2  Performance  Assessment  and  Root  Cause  Analysis  Office,  2013. 

3  Blickstein  et  al.,  2011. 
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Table  1.1 
FA  Areas 


Technological 

(Component/System 

Integration) 

Management/ 
Program  Structures 

Mission  Requirements 

Cost  and  Schedule 
Expectations 

Manufacturing 

Dependencies  on  other 

Stability  of  operational 

Industrial  base/market 

expectations 

programs  or 
development  efforts 

needs 

Quantity 

expectations 

Testing  expectations 

Capabilities 

Acquisition  initiatives  or 

Technical  approach 

Contractual  incentive 
strategy/relationships 

Joint  needs 

Possibility  of  a  substitute 

targets 

Unknown  or  undefined 

Risk  expectations 

Organizational 
management  structure 

system 

areas  of  scope  (e.g., 
facilities  locations. 

Use  of  simulation 

Legal,  diplomatic,  or 

Understanding  of  threat 
levels 

support  approaches) 

Scale  of  integration 

political  issues 

Flexibility  based  on 

Experience  of  industry 
to  execute 

COTS/GOTS  suitability  for 
application 

Reusability  of  legacy 
equipment  or  subsystems 

Technical  maturity  of 
components,  system,  and 
software 

Degree  of  "Jointness" 

FMS  possibilities 

changing  intelligence 

-  The  commonality  between  variants  is  high  and  will  reduce  development  and 
production  cost. 

•  Evolved  Expendable  Launch  Vehicle  (EELV) 

-  The  commercial  launch  marketplace  is  robust  and  can  be  leveraged  for  savings. 

-  Government  launch  systems  are  flexible. 

•  Space  Based  Infrared  System  (SBIRS)  High 

-  Total  Systems  Performance  Responsibility  (TSPR)  approach,  managed  by  the 
contractor,  is  more  effective  than  a  traditional  government-led  management 
approach. 

-  Software  can  be  reused  across  development  increments. 

-  A  form  of  incremental  funding  will  yield  large  cost  savings. 

It  is  important  to  note  that  FAs  are  not  necessarily  a  “bad”  thing  or  incorrect.  All 
programs  make  assumptions  and  many  may  turn  out  to  be  valid.  The  important  point 
is  that  program  outcomes  are  highly  dependent  on  making  the  correct  FAs.  Also,  one 
could  go  overboard  in  identifying  framing  assumptions.  The  concept  is  to  develop  a 
few  concrete,  high-level  assumptions  that  can  be  easily  tracked;  on  the  order  of  three 
to  five  assumptions  are  a  good  amount  for  most  programs. 
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The  Three  Phases  of  FA  Identification 

The  process  for  identifying  FAs  involves  three  phases  structured  to  occur  in  a  linear 
manner:  nomination,  validation,  and  prioritization. 

Phase  1:  Candidate  FA  Nomination 

In  the  nomination  phase,  candidate  FAs  are  identified.  This  phase  is  meant  to  be  one  of 
idea  generation,  in  which  no  candidate  FA  that  is  nominated  is  immediately  dismissed. 
The  goal  is  to  obtain  a  robust  list  that  will  later  be  pared  down  using  a  set  of  eligibility 
criteria.  This  method  is  also  known  as  brainstorming.  It  is  important  to  have  a  mix  of 
different  people  with  different  backgrounds  participate  (e.g.,  technologist,  implement- 
ers,  managers)  so  that  a  wide  variety  of  ideas  are  put  forward  during  the  nomination 
phase. 

Phase  2:  Candidate  FA  Validation 

In  the  validation  phase,  each  candidate  FA  that  was  nominated  is  validated  against 
three  of  the  four  criteria  defined  in  PARCAs  Information  Paper,4  namely,  whether  each 
candidate  FA  is  foundational,  program-specific,  and  without  workarounds.  Candidate 
FAs  are  eliminated  if  they  do  not  meet  any  one  of  the  three  criteria.  The  goal  is  to  deter¬ 
mine  the  eligibility  of  those  assumptions  nominated  as  candidate  FAs.  The  candidate 
FAs  that  have  been  validated  become  the  program  s  potential  FAs. 

Phase  3:  FA  Prioritization 

In  the  prioritization  phase,  each  remaining  potential  FA  is  prioritized  by  its  criticality. 
The  goal  of  this  phase  is  to  rank  the  potential  FAs  to  inform  the  selection  of  three  to 
five  FAs  for  the  program.  The  program  manager  will  likely  make  this  selection  of  the 
final  FAs. 


Operationalizing  the  Approach  Using  a  Structured  Deliberation 
Process 

The  three  phases  outlined  in  the  previous  section  are  operationalized  using  a  five-step 
structured  group  deliberation  process.  This  section  provides  supporting  information 
related  to  that  process. 

Why  a  Structured  Approach? 

The  deliberation  process  set  out  in  the  briefing  was  selected  for  a  number  of  reasons. 
First,  a  structured  process  allows  the  group  to  avoid  some  of  the  common  drawbacks 


Performance  Assessment  and  Root  Cause  Analysis  Office,  2013. 
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of  open-ended  group  idea-sharing  processes.  Since  group  members  use  the  delibera¬ 
tion  time  both  to  generate  their  own  ideas  and  to  process  the  ideas  of  others,  the  two 
activities  may  compete  with  one  another  for  a  group  member  s  attention  and  limit  the 
beneficial  effects  of  idea-sharing.  To  overcome  this  constraint,  a  structured  process  can, 
for  instance,  promote  techniques  for  sharing  written  ideas.  By  having  group  members 
write  down  their  ideas  ahead  of  a  group  discussion,  they  do  not  have  to  wait  their 
turn  to  generate  and  record  ideas.5  Writing  down  ideas  ahead  of  time  should  not  be 
construed  as  precluding  group  members  from  freely  commenting  on  other  members’ 
contributions  to  the  deliberation  or  from  considering  new  ideas  during  the  discussion. 
Both  of  these  activities  are  encouraged  as  a  part  of  the  deliberation.  Rather,  initially 
writing  down  ideas  will  allow  group  members  to  generate  their  own  initial  ideas  with¬ 
out  the  distractions  that  can  occur  during  the  group  discussion.  This  activity  is  also 
helpful  for  mitigating  a  second  drawback  of  unstructured  group  idea-sharing — that 
relevant  information  held  by  a  minority  of  group  members  often  fails  to  be  introduced 
and  given  equal  weight  in  the  conversation. 

A  second  motive  for  using  a  structured  process  is  that  it  allows  for  better  repro¬ 
ducibility,  easier  documentation,  and  more  control.  Although  the  candidate  FAs  that 
are  originally  nominated  could  differ  if  the  group  member  composition  were  to  change, 
the  structured  process  will  likely  drive  the  final  FAs  to  be  quite  similar.  The  structure 
also  allows  for  a  simple  table  (the  FA  Matrix)  to  act  as  documentation  of  the  process. 
Finally,  a  structured  process  provides  a  facilitator  with  more  control  to  consider  minor¬ 
ity  viewpoints  and  explore  all  ideas. 

Selecting  Participants 

The  proper  selection  of  participants  for  the  group  deliberation  process  is  essential  to  the 
success  of  the  process.  Choosing  a  heterogeneous  group  of  participants,  with  diverse 
backgrounds  and  a  broad  range  of  knowledge  about  the  program,  will  ensure  that  all 
candidate  FAs  are  considered.  Furthermore,  having  a  diverse  group  may  reduce  the 
tendency  for  the  group  to  exhibit  “confirmation  bias”  or  the  inclination  to  seek  out 
only  the  information  that  confirms  a  specific  viewpoint  (and  disregard  information 
that  disconfirms  it).6  Confirmation  bias  could  lead  to  the  incorrect  elimination  of  an 
FA  or  validation  of  an  ineligible  FA. 

Selecting  a  Facilitator 

The  group  facilitator  will  play  an  important  role  in  the  group  deliberation  process — 
one  of  neutral  leadership.  Thus,  he/she  does  not  contribute  to  the  content  of  the  dis¬ 
cussion  or  evaluate  any  ideas  offered  by  group  members  but  rather  ensures  that  the 
content  is  within  the  proper  scope  of  the  discussion.  The  facilitator  fosters  the  process 


5  Paulus  and  Yang,  2000. 

6  Straus,  Parker,  and  Bruce,  2011. 
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of  group  deliberation  by  keeping  the  discussion  on  track  and  making  suggestions  on 
how  to  proceed.  He/she  also  ensures  that  everyone  in  the  group  has  a  chance  to  speak 
and  that  all  ideas  expressed  by  group  members  are  included  in  the  deliberation  pro¬ 
cess  and  not  immediately  dismissed.  While  performing  a  leadership  role,  the  ultimate 
goal  of  any  facilitator  should  be  that  the  group  members  could  perform  their  delibera¬ 
tion  without  him/her.  That  is,  the  facilitator’s  role  should  be  a  passive  one  in  the  best 
scenario.  Overall,  a  facilitator  must  strike  a  delicate  balance  between  supporting  the 
group  deliberation  process  and  not  dictating  its  direction. 

Choosing  a  facilitator  for  the  group  discussion  helps  to  ensure  that  the  structure 
set  out  in  the  briefing  is  followed.  It  also  allows  for  a  neutral  voice  that  can  mediate  dis¬ 
agreements,  ensure  that  the  minority  viewpoint  is  considered,  and  discourage  discus¬ 
sion  that  is  off  task.  The  benefit  of  using  a  facilitator  depends  on  the  person  selected  to 
lead.  That  is,  if  the  facilitator  has  a  stake  in  the  final  outcome  of  the  deliberation  or  has 
a  strong  predisposition,  he/she  may  bias  the  result  of  the  process.  For  this  reason,  it  is 
advised  that  the  program  manager  (PM)  not  be  chosen  as  the  facilitator.  For  instance, 
an  appropriate  choice  of  a  facilitator  may  be  someone  who  is  not  a  part  of  the  program 
office  and  therefore  does  not  have  a  stake  in  the  process. 

Outline  of  Presentation 

The  main  briefing  (Slides  1-27)  present  an  overview  of  the  structured  deliberation  pro¬ 
cess  and  walks  through  the  five  steps  in  the  process.  Slides  1-5  introduce  the  process 
and  gives  an  overview  of  the  meeting  objectives.  Step  1  (Homework)  takes  place  before 
the  meeting  where  participants  familiarize  themselves  with  the  concept  of  FAs  and 
consider  potential  assumptions  for  their  program.  Also,  a  program  overview  should 
be  provided  to  participants  who  are  not  actively  working  on  the  program  to  be  read 
as  a  homework  assignment  before  the  deliberations.  Alternatively,  the  PM  could  pro¬ 
vide  an  overview  briefing  at  the  beginning  of  the  session.  Step  2  (Slides  7-13)  reviews 
the  concept  of  FAs  and  provides  some  case  study  examples  for  three  programs  (Slides 
11-13).  These  case  study  examples  can  be  tailored  to  suit  the  specific  program  being 
reviewed.  Additional  examples  are  provided  as  backup  (Slides  41-44).  Step  3,  identi¬ 
fying  candidate  FAs,  is  presented  in  Slides  14-17  Step  4  (Slides  18-23)  screens  can¬ 
didate  FAs  against  the  criteria  defined  by  PARC  A  (see  above).  The  final  step,  Step  5 
(Slides  24-27),  examines  the  surviving  FAs’  (now  potential  FAs)  criticality  and  identi¬ 
fies  potential  tracking  metrics.  Slide  28  includes  a  blank  template  of  the  FA  Matrix  that 
the  facilitator  can  use  to  document  the  deliberation  process. 

Slides  30-44  are  backup  slides.  Slide  30  lists  supporting  references  for  the 
material  in  case  the  team  wants  to  explore  specific  concepts  in  more  depth.  Slides  31-37 
present  a  series  of  seed  questions  meant  to  stimulate  discussion  on  unique  or  unusual 
aspects  of  the  program.  Answering  “yes”  to  one  of  the  questions  does  not  necessarily 
mean  that  there  is  an  FA  for  the  program.  The  questions  cover  the  four  major  themes 
for  FAs  so  that  all  themes  are  considered.  These  questions  could  be  tailored  or  modi- 
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fied  depending  on  the  program.  For  example,  questions  about  production  methods  or 
technology  may  not  be  applicable  to  programs  that  are  mainly  software/information 
technology  in  nature.  Slide  38  lists  a  series  of  additional  questions  to  explore  when  con¬ 
sidering  candidate  FAs.  Slide  39  presents  supporting  questions  when  considering  FA 
criticality.  Slides  40-44  provide  additional  case  study  FA  examples. 


CHAPTER  TWO 


Slides 


In  this  section,  we  reproduce  the  slides  from  the  presentation. 


NATIONAL  DEFENSE  RESEARCH  INSTITUTE 


Identifying  and  Prioritizing  Acquisition 
Framing  Assumptions:  Structured 
Deliberation  Exercise 


July  2014 
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Group  Deliberation  Goals 


•  Objective 

-  Identify  and  prioritize  a  set  of  major  program 
assumptions 

•  Three  phases: 


FA  Nomination  FA  Validation  FA 

^^_^B  Prioritization 


’ - N 

•  Identify 
candidate 

FAs 

V _ y 

■ 

r - n 

•  Determine 
candidate 

FA  eligibility 

L  J 

m 

r  \ 

•  Rank  FAs 
by  criticality 

v  J 
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The  Group  Deliberation  Exercise 

•  Small  interactive  group  exercise  to  stimulate  creative 
idea  generation 

•  Participants  should: 

-  Voice/record  as  many  ideas  as  possible  (regardless 
of  their  feasibility  or  unusual  nature) 

-  Withhold  criticism  or  initial  evaluation  of  ideas 

-  Extend,  add  onto,  combine  and  build  upon  other 
participants’  ideas 

•  Some  methods  are  better  than  others 

-  Unrestricted  group  deliberation  has  been  shown  to 
be  less  effective  than  structured,  written  idea 
sharing 

Sources:  Paulus  and  Yang,  2000;  DeRosa,  Smith,  and  Hantula,  2007. 
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Ground  Rules  for  a  Deliberation  Session 

*  Follow  a  semi-structured,  systematic  process 

*  Appoint  a  facilitator  or  group  leader  who  does  not  have  a  stake 
in  the  outcome  or  strong  predispositions;  the  facilitator  should 
NOT  be  the  program  manager 

*  Gather  a  small,  heterogeneous  group 

*  Allow  enough  time  in  the  session  to  properly  explore  all  ideas 

*  Never  immediately  dismiss  a  new  idea/minority  viewpoint  - 
understand  its  basis  and  potential  consequences 

*  Keep  an  open  mind  and  be  willing  to  change  initial  opinions 

*  Use  methods  that  allow  all  group  members  to  state  their  opinion 
such  as  the  sharing  of  individually  written  ideas 

Sources:  Straus,  Parker,  and  Bruce,  2011;  Central  Intelligence  Agency, 

2009;  DeRosa,  Smith,  and  Hantula,  2007. 


RAND 


Framing  Assumptions  Structured  Deliberation  Exercise  -  4 

Jan  2013 


Slides  13 


Five  Steps  in  the  Deliberation  Process 


•  Step  1:  “Homework”:  Read  Informational  Material 
(required),  identify  3+  candidate  FAs  (optional) 

•  Step  2:  Overview/Review  of  FAs  and  their  importance 

•  Step  3:  Identify  candidate  FAs  ( Nomination ) 

•  Step  4:  Determine  eligibility  ( Validation ) 

•  Step  5:  Rank  by  criticality  ( Prioritization );  identify 
metrics  to  track 
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Step  1 :  “Homework” 

Read  FA  Info  Paper  and  Program  Background 
(required);  Identify  3+  Candidate  FAs  (optional) 


©  Read  FA  Info  Paper  and  FA  questions  (complete  set 
in  back-up  slides)  prior  to  meeting  for  deliberation 

(2)  Read  program  background 

CD  Group  members  come  prepared  to  meeting  with  3+ 
candidate  FAs  (optional) 
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Step  2:  Overview  of  Framing  Assumptions 
and  Their  Importance 


©  Definition  of  an  FA  and  common  FA  themes 
(2)  Importance  of  FAs 
CD  Program  example  FAs 
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Definition  of  a  “Framing  Assumption” 

•  Framing  assumption  definition:  Any  explicit  or  implicit  assumptions  that  are  central 
in  shaping  cost,  schedule,  and/or  performance  expectations 

•  Attributes  of  a  framing  assumption: 

-  Critical:  Assumption  significantly  affects  program  expectations  (e.g.,  cost,  schedule, 
performance). 

-  No  workarounds:  The  consequences  of  an  incorrect  FA  cannot  be  easily  mitigated.  The 
effects  are  generally  outside  the  project  team’s  control. 

-  Foundational:  Not  subordinate,  derivative  or  linked  to  other  assumptions.  The  FA  may 
be  composed  of  secondary  assumptions,  but  it  is  independent  from  other  major 
assumptions.  It  represents  some  central  feature  of  the  program. 

-  Program  specific:  Not  generically  applicable  to  all  programs. 

•  Only  a  few,  key  framing  assumptions  per  program 


A  Framing  Assumption  is... 

A  Framing  Assumption  is  not... 

A  unique  aspect  of  contracting  strategy  (e.g.,  competitive 
prototyping) 

The  contractor  will  perform  well 

Weapon  system  to  be  replaced  will  last  until  a  specified 
time 

Program  characteristic  (e.g.,  family  of  systems) 

Use  of  COTS/GOTS  subsystems  will  save  money 

Program  is  affordable 

Framing  assumptions  are  not  necessarily  incorrect  or  “bad.”  However,  they  fundamentally  shape 
program  success  -  a  bad  assumption  can  result  in  significant  challenges. 
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Common  Framing  Assumption 


Technological 
(Component  /  System 
Integration) 


•  Manufacturing 
expectations 

•  Testing  expectations 

•  Technical  approach 

•  Risk  expectations 

•  Use  of  simulation 

•  Scale  of  integration 

•  COTS/GOTS 
suitability  for 
application 

•  Reusability  of  legacy 
equipment  or 
subsystems 

•  Technical  maturity  of 
components, 
system,  and 
software 


Management/ 
Program  Structures 


Dependencies  on 
other  programs  or 
development  efforts 
Contractual 
incentive  strategy/ 
relationships 
Organizational 
management 
structure 

Legal,  diplomatic,  or 
political  issues 
Degree  of 
“Jointness” 

FMS  possibilities 


Mission 

Requirements 


•  Stability  of 
operational  needs 

•  Quantity 

•  Capabilities 

•  Joint  needs 

•  Possibility  of  a 
substitute  system 

•  Understanding  of 
threat  levels 

•  Flexibility  based  on 
changing 
intelligence 


Themes 


Cost  and  Schedule 
Expectations 


Industrial  base/ 
market  expectations 
Acquisition  initiatives 
or  targets 
Unknown  or 
undefined  areas  of 
scope  (e.g.,  facilities 
locations,  support 
approaches) 
Experience  of 
industry  to  execute 
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Importance  of  F As:  Failed  FAs  Have  Been  Major 
Contributors  to  Nunn-McCurdy  Breaches 

•  Joint  Strike  Fighter  (F-35) 

-  Competitive  prototypes  are  production  representative 

-  The  commonality  between  variants  is  high  and  will  reduce 
development  and  production  cost 

•  EELV 

-  The  commercial  launch  marketplace  is  robust  and  can  be 
leveraged  for  savings 

•  Space  Based  Infrared  System  (SBIRS)  High 

-  Total  Systems  Performance  Responsibility  (TSPR)  approach, 
managed  by  the  contractor,  is  more  effective  than  traditional 
government-led  approach 

-  Software  can  be  reused  between  increments 

-  A  form  of  incremental  funding  will  yield  large  cost  savings 


RAND 


Framing  Assumptions  Structured  Deliberation  Exercise  - 10 

Jan  2013 


Slides  19 


Advanced  Pilot  Training  (APT) 
Family  of  Systems  (FoS) 


Service:  Air  Force 


Program  Type:  Pre-MDAP 

Commodity  Type:  Aircraft  and  Ground  Training  System 


Description:  Expected  to  supply  an  advanced  trainer,  known  as  the  T-X  aircraft,  for 
the  fighter/bomber  APT  track  as  soon  as  2017 


Technological 


•  Training  aircraft 
will  be  non- 
developmental 
(leverage 

commercial/foreign 
aircraft’s 
capabilities)  with 
minimal  changes. 


Management/ 
Program  Structures 


•  Possible  synergies 
with  U.S.  Navy  and 
foreign  militaries, 
(requirements 
align  and  minimal 
modifications 
required). 


Mission 

Requirements 


•  Use  of  existing 
T-38  can  be 
extended  until 
2020. 


Cost  and  Schedule 


•  Simulators  can  be 
used  instead  to 
reduce  training 
flight  time  (to  save 
money). 
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Joint  Lightweight  Tactical  Vehicle  (JLTV) 


Service:  DoD  (Army  and  Marine  Corp) 
Program  Type:  Pre-MDAP 
Commodity  Type:  Ground  Combat 


Description:  Will  be  a  successor  to  the  11  different  versions  of  the  High  Mobility, 
Multi-Wheeled  Vehicle  (HMMWV)  that  have  been  in  service  since  1985 


Technological 


•  Open  architecture 
approach  will 
reduce  risk  and 
allow  for  more 
efficient  upgrades. 


Management/ 
Program  Structures 


•  Joint  Army  and 
Marine  Corps 
program  will  save 
money  and 
requirements  are 
compatible. 


Mission 

Requirements 


•  Have  effectively 
assessed  long¬ 
term  vs.  short-term 
needs. 


Cost  and  Schedule 


•  Competitive 
prototyping  will 
reduce  risk/cost. 
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Joint  Precision  Approach  and  Landing  System 

(JPALS) 

Service:  Navy  (with  Air  Force  and  Army  support) 

Program  Type:  MDAP 

Commodity  Type:  Command  and  Control  (Other) 

Description:  Global  Positioning  System  (GPS)-based  precision  approach  and  landing 
system 


Technological 


•  Modeling  and 
simulation  can  be 
used  to  explore 
design  tradeoffs. 


Management/ 
Program  Structures 


•  Navy  is  best 
service  to  lead 
acquisition  due  to 
their  more 
demanding 
requirements. 


Mission 

Requirements 


•  GPS  constellation 
will  be  robust. 

•  System  is  suitable 
for  all  types  of  air 
vehicles. 

•  FAA  will  move  to  a 
GPS-based  ILS. 


Cost  and  Schedule 


•  Ability  to  leverage 
COTS/GOTS 
hardware  and 
software  will  lower 
cost. 
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Step  3:  Identify  Candidate  Framing  Assumptions 


©  Group  members  identify  and  individually  record 
candidate  FAs 

(D  Facilitator  lists  candidate  FAs  in  FA  Matrix ;  group 
members  present  their  FAs  to  the  group 

(3)  Facilitator  leads  limited  discussion 
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(T)  Group  Members  Identify  and  Individually 
Record  Candidate  FAs 

*  Group  members  read  seed  questions  -  questions  meant  to  raise  issues  that 
may  lead  to  candidate  FAs 

*  Members  identify  and  record  candidate  FAs,  along  with  a  brief  supporting 
explanation  for  each  (these  may  be  prepared  ahead  of  time  if  Step  1  is 
implemented  fully) 

*  The  seed  questions  are  drawn  from  each  common  FA  theme;  also  include 
more  general  questions  to  help  identify  assumptions;  examples  include 
(see  back-up  slides  for  complete  set): 

-  Technical:  Have  the  technologies/types  of  software  employed  been  used  in 
a  similar  application,  scale,  or  environment? 

-  Management:  Is  the  program’s  progress  dependent  on  the  progress  of  other 
programs?  Is  a  new  program  management  approach  being  adopted? 

-  Requirements:  Are  the  requirements  stable  and  well  defined? 

-  Cost/schedule:  Are  there  any  significant  savings  initiatives/targets  assumed? 
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2)  Facilitator  Lists  Candidate  FAs  in  FA  Matrix; 
Members  Present  Their  FAs  to  the  Group 


•  Facilitator  compiles  all  candidate  FAs  and  lists  in  the  first 
column  of  the  FA  Matrix,  eliminating  redundant  FAs 

*  Members  briefly  present  their  candidate  FAs  to  the  group 


The  FA  Matrix 


Candidate 

FA 

Eligibility 

Prioritize 

Criticality 

Signposts/ 

Program- 

specific? 

No  work¬ 
arounds? 

Foundational? 

metrics  to 
monitor  FA 

FA  1 

fMMMS 

1 

1  1 

FA  2 

;||| 

WMMMM, 

!!ll 

: 

: 

FAn 
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(3)  Facilitator  Leads  Limited  Discussion 


*  Facilitator  leads  a  limited  discussion  to  clarify  or  add 
additional  candidate  FAs 

*  Facilitator  should  ensure  that  all  common  FA  themes  are 
considered  when  compiling  list  of  candidate  FAs 

-  Any  missing  FA  themes  should  be  revisited 

*  At  this  time  group  members*: 

-  May  ask  for  clarification  and  rewording  of 
candidate  FAs 

-  Add  any  new  candidate  FAs  that  are  uncovered 
during  discussion 

*Group  members  should  not  discuss  whether  the  candidate  FA  is 
foundational,  program-specific,  critical,  or  has  no  workarounds 
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Step  4:  Determine  Candidate  FA  Eligibility 


©  Are  candidate  FAs  program-specific? 

(D  Do  candidate  FAs  have  any  workarounds? 
(D  Are  candidate  FAs  foundational? 
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(T)  Are  Candidate  FAs  Program-Specific? 

*  Facilitator  leads  a  discussion  about  whether  each  candidate  FA  is  program- 
specific? 

-  The  assumption  is  not  generally  applicable  to  all  programs 

-  The  assumption  is  not  a  fundamental  physical  law  or  property  -  such  as  the 
gravitational  constant. 

*  Group  comes  to  consensus  on  whether  each  candidate  FA  is  program-specific 

*  If  candidate  FA  is  determined  to  not  be  program-specific,  it  is  eliminated  from 


eligibility 
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Candidate 

FA 

\  Eligibilit 
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Prioritize 

Criticality 
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Yes 
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No 
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: 
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(£)  Do  Candidate  FAs  Have  Any  Workarounds? 

*  Facilitator  leads  discussion  about  whether  each  candidate  FA  has  any  workarounds; 
questions  include: 

•  Could  any  of  the  consequences  be  avoided  through  some  other  means? 

•  Could  the  success  of  the  program  still  be  salvaged  through  some  mitigating 
action? 

*  Group  comes  to  consensus  on  whether  each  candidate  FA  has  any  workarounds 

*  If  candidate  FA  is  determined  to  have  workarounds,  it  is  eliminated  from  eligibility 
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FA 

Eligibility 
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Foundational? 
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wmm 

FA  2 

No 

: 

Yes 

Yes 

: 
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(3)  Are  Candidate  FAs  Foundational? 


•  Record  all  remaining  candidate  FAs  on  self-adhesive  notes 

•  With  minimal  talking,  move  candidate  FAs  around  into  groupings 

-  Groupings  should  reflect  some  aspect  of  similarity  between  FAs 

-  Candidate  FAs  may  be  moved  more  than  once  by  different  individuals 

-  If  there  is  too  much  back-and-forth  on  a  candidate  FA,  create  a  copy  of  it  and  place  it  in  two 
categories 

•  For  each  grouping  and  remaining  lone  candidate  FAs,  identify  other  relevant  candidate  FAs: 

-  Are  there  related  assumptions  that  have  not  been  identified  yet? 

-  Is  there  a  new  candidate  FA  that  is  central  to  all  others  in  the  grouping? 

•  Identify  the  foundational  FA: 

-  Are  there  similarities  within  a  group? 

-  Are  some  candidate  FAs  subordinate  or  a  derivative  of 
other  FAs? 

-  Is  the  candidate  FA  or  grouping  contingent  on  other 
events? 

-  Does  the  candidate  FA  or  grouping  represent  one  or  more 
aspects  of  a  more  general  assumption? 

Source:  Institute  for  Healthcare  Improvement ,  2004.  Green  notes:  Candidate  FAs 

Blue  notes:  foundational  FAs 
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(3)  Example  Foundational  and  Subordinate 

Candidate  FAs 

Littoral  Combat  Ship  Modules  (Navy  mission  systems 
program  developing  modules  to  execute  a  variety  of 
missions) 

*  Candidate  FAs  that  may  be  grouped  together: 

-  Incremental  development  will  allow  new  capabilities 
to  be  added  easily 

-  Modules  are  able  to  be  tested  on  other  ship 
platforms 

*  Foundational  FA:  Mission  modules  can  be  developed 
independently  of  sea  frames 
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FA  Matrix  with  All  Eligibility  Items 

_ n _ 
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Step  5:  Prioritize  Potential  FAs  by  Criticality;  Identify 

Metrics  to  T rack 


©  Determine  criticality  of  potential  FAs 

(D  Determine  signposts/metrics  to  monitor  candidate 
FAs 
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(2)  Determine  Signposts/Metrics  to  Monitor 

Potential  FAs 

•  Facilitator  leads  discussion  about  signposts  and  metrics 
that  could  be  used  by  the  program  to  monitor  the 
assumption;  questions  include: 

-  What  events  or  trends  (i.e.,  signposts)  would  be  expected  to 
occur  if  this  assumption  was  correct  or  incorrect? 

•  How  could  these  events/trends  be  measured  and 
monitored? 

-  What  metrics  could  be  used  to  track  the  validity  of  the  FA? 

•  Are  there  threshold  values  for  these  metrics  that  could 
signify  a  change  in  the  assumption? 

*  Facilitator  records  ideas  on  the  FA  Matrix 


Source:  Central  Intelligence  Agency,  2009. 
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0  Determine  Signposts/Metrics  to  Monitor 

Potential  FAs 

•  Facilitator  leads  discussion  about  signposts  and  metrics 
that  could  be  used  by  the  program  to  monitor  the 
assumption;  questions  include: 

-  What  events  or  trends  (i.e.,  signposts)  would  be  expected  to 
occur  if  this  assumption  was  correct  or  incorrect? 

•  How  could  these  events/trends  be  measured  and 
monitored? 

-  What  metrics  could  be  used  to  track  the  validity  of  the  FA? 

•  Are  there  threshold  values  for  these  metrics  that  could 
signify  a  change  in  the  assumption? 

*  Facilitator  records  ideas  on  the  FA  Matrix 


Source:  Central  Intelligence  Agency,  2009. 
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An  Example  of  a  Completed  FA  Matrix 
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FA  Matrix  Template 
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Seed  Questions 


The  next  set  of  slides  include  the  following  seed  questions 

1.  Topical  questions  to  help  identify  assumptions 

-  Following  common  themes:  Technical, 

Management,  Requirements,  Schedule/Cost 

-  Answering  yes  to  the  question  might  indicate  a 
possible  FA  for  the  program  in  that  area.  The 
questions  should  be  tailored  to  the  program 

2.  General  questions  to  help  identify  assumption 

3.  Questions  to  assess  criticality 
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Technical  Discussion  Questions 


•  Have  the  technologies  employed  been  used  in  a  similar  application  or  environment? 

-  Is  there  commercial  technology  that  is  being  used  for  the  first  time  in  a  military 
application?  Who  has  the  data  rights? 

-  Has  the  technology  demonstrated  successfully  under  the  same  operating 
conditions? 

-  Has  the  reliability  been  demonstrated? 

•  Does  the  system  depend  on  COTS  solutions  or  other  commercial  technologies  and 
services? 

-  Is  this  a  novel  integration  of  standard  systems? 

-  Will  these  systems  require  modification  for  environment  (e.g.,  shock,  vibration, 
electromagnetic,  and  temperature)? 

-  How  long  might  the  manufacturer  support  such  an  item?  Will  these  services  be 
available  over  the  life  of  the  system?  Can  the  design  adapt  to  component 
changes/upgrades? 

•  Is  the  commercial  availability  stable?  Have  all  the  technologies  been  demonstrated  or 
successfully  operated  at  the  scale  planned  (e.g,.  power  density,  number  of  sensors, 
bandwidth)? 

-  How  large  a  scale-up  is  planned? 

-  Are  there  integration  issues  at  this  scale? 
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Technical  Questions  (cont.) 


*  Are  there  multiple  systems/family  of  systems  (FoS)  that  need  to  be  integrated? 

-  Have  similar  FoS  been  successfully  integrated? 

*  Are  there  new  manufacturing  methods  or  techniques  involved? 

*  Are  there  new  or  unusual  materials  involved? 

-  Is  the  source  of  supply  and  price  stable? 

*  Have  prototypes  been  developed  (or  are  planned)  at  the  subcomponent  or  system  level? 

-  Do  the  prototypes  represent  something  close  to  a  production  configuration? 

-  Has  the  prototype  effort  resulted  in  reductions  to  cost  and/or  schedule? 

-  Have  the  prototypes  demonstrated  needed  technical  maturity? 

*  Is  the  software  development  well  understood? 

-  What  is  the  size  of  the  overall  programming  effort? 

-  How  many  lines  of  code  per  day  can  we  expect?  What  error  rate? 

-  What  are  the  assumptions  around  software  reuse? 


RAND 


Framing  Assumptions  Structured  Deliberation  Exercise  -  33 

Jan  2013 


42  Identifying  Acquisition  Framing  Assumptions  Through  Structured  Deliberation 


Management  Questions 

*  Novel  management  structures 

-  Is  the  government  acting  as  system  integrator? 

-  Are  multiple  PEOs/PMs  involved? 

-  Do  industry  partners  participate  through  new  commercial 
partnerships  or  JVs? 

*  Is  the  program’s  progress  dependent  on  the  progress  of  other 
programs? 

*  Are  there  unique  legal,  diplomatic,  or  security  issues? 

*  Does  the  program  have  an  experienced  workforce?  Will  there  be 
issues  retaining  this  workforce? 
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Requirements  Questions 

*  Is  there  joint/foreign  involvement? 

-  Are  the  program  requirements  compatible  between  the  stakeholders? 

-  Does  each  participant  require  a  customized  version? 

-  Is  there  uncertainty  with  respect  to  quantities  for  partners? 

*  Are  the  requirements  stable  and  well  defined? 

*  Will  capability  be  met  through  an  evolving  design  or  series  of  upgrades? 

*  Is  the  need  well  understood  (both  from  a  capability  sense  and  timing)? 

*  Are  there  unknown  major  areas  of  scope,  e.g.,  facilities  locations,  operational 
availability,  support  equipment/infrastructure? 

*  Could  another  system  substitute  for  this  one? 

*  Do  all  the  requirements  need  to  be  addressed  for  the  program  to  be 
successful? 


RAND 


Framing  Assumptions  Structured  Deliberation  Exercise  -  35 

Jan  2013 


44  Identifying  Acquisition  Framing  Assumptions  Through  Structured  Deliberation 


Cost  and  Schedule  Questions 


•  Does  the  program  rely  on  sole  source(s)? 

•  Have  the  intellectual  property  rights  been  resolved? 

•  Are  there  workforce  supply  or  demand  issues?  For  instance,  will  the  program 
contractor  or  vendor  require  significant  hiring?  Are  key  workforce  skills/ 
trades  in  short  supply?  Can  we  hire  at  the  rate  based  in  our  plans  for  that 
location? 

•  Is  the  stability  of  the  vendors  and  suppliers  base  understood?  Are  there  key 
suppliers  who  are  at  risk? 

•  Has  the  prime  contractor  executed  a  similar  program  (either  in  complexity  or 
system/commodity  type)  before? 

•  Is  there  going  to  be  a  management  reserve/should  there  be  cost  targets  for  the 
program? 

•  Has  sufficient  time  been  allowed  to  get  the  necessary  approvals  from  OSD? 
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Cost  and  Schedule  Questions  (cont.) 


•  In  the  case  of  COTS  products/solutions  -  is  the  commercial  marketplace 
stable  in  terms  of  demand  and  pricing  or  is  it  cyclical?  Can  you  obtain  the 
data  needed  from  the  COTS  supplier? 

*  Are  there  any  significant  savings  initiatives/targets  assumed? 

-  What  is  the  source  of  the  savings? 

-  Are  they  reasonable  based  on  experience  on  existing  or  completed 
programs? 

•  Are  there  schedule  pressures  or  tight  deadlines  to  meet  an  IOC  date? 

*  Is  the  testing  plan  adequate? 

-  Is  the  time  allotted  comparable  to  previous  efforts? 

-  Is  the  number  of  test  articles  similar? 

-  Is  there  a  reliance  on  simulation  to  supplant  some  of  the  testing? 

-  Does  the  test  community  have  the  capacity  in  the  timeframe  needed? 

-  Is  there  particular  test  equipment  needed?  Does  it  need  to  be 
developed? 


RAND 


Framing  Assumptions  Structured  Deliberation  Exercise  -  37 

Jan  2013 


46  Identifying  Acquisition  Framing  Assumptions  Through  Structured  Deliberation 


General  Questions  to  Identify  Assumptions 

Consider  the  following  questions  when  determining 
potential  framing  assumptions  from  seed  questions 

*  Does  information  exist  that: 

-  Disconfirms  or  is  contradictory  to  this  judgment? 

-  Was  previously  dismissed  that  might  now  be  relevant  to  the  topic? 

-  Is  new  and  could  change  this  judgment?  Has  it  been  properly  adjusted? 

*  How  accurate  and  reliable  is  information  upon  which  this  judgment  is  based? 

-  Was  incomplete,  imprecise,  or  ambiguous  information  used? 

*  Could  certain  circumstances  (e.g.,  social,  technological,  economic, 
environmental,  political,  organizational)  affect  this  judgment? 

-  Does  the  judgment  account  for  these  circumstances?  How  sensitive  is  it 
to  these  circumstances? 

-  Could  circumstances  proceed  differently  than  expected? 

-  For  what  circumstances  would  this  judgment  be  abandoned? 

-  Have  all  plausible  but  unpredictable  circumstances  been  considered? 

Sources:  Straus,  Parker,  and  Bruce,  2011;  CIA  2009;  Kebell,  Muller,  and  Martin,  2010. 
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Criticality  Questions 

•  If  this  assumption  was  incorrect: 

-  To  what  degree  would  it  affect  the  success  of  the  program? 

•  Could  it  delay  the  program  schedule? 

•  Could  it  change  program  requirements? 

•  Could  it  affect  program  performance? 

•  Could  it  directly  increase  program  costs? 

-  Could  this  result  in  other/multiple  consequences  to  the 
program? 

*  Would  changes  in  this  assumption  result  in  the  contradiction  of 
other  identified  assumptions? 


Source:  Central  Intelligence  Agency,  2009. 
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Further  Examples 
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Littoral  Combat  Ship  (LCS)  Mission  Modules 

•  Service:  Navy 

•  Program  Type:  Pre-MDAP 

•  Commodity  Type:  Mission  Systems 

•  Description: 

-  Provides  Combatant  Commanders  assured  access  against 
littoral  threats 

-  Mission  systems  are  added  to  the  mission  module  baseline 
incrementally  as  they  reach  a  level  of  maturity  necessary  for 
fielding 

-  Uses  evolutionary  acquisition  process 

-  Provides  an  open  architecture  environment  that  enables  future 
rapid  insertion  of  new  technologies 

•  Last  Milestone  Awarded:  Milestone  A  (May  27,  2004) 

Sources:  DAES  (as  of  April  17,  2012)  (FOUO);  “LCS  Mission  Modules:  Training  Strategy  Increasing 
Modularity  for  Maximum  Adaptability  Brief  for  Implementation  Fest  2010”  (August  10,  2010) 
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Littoral  Combat  Ship  (LCS)  Mission  Modules 
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Space  Fence 

•  Service:  Air  Force 

•  Program  Type:  Pre-MDAP 

•  Commodity  Type:  Radar 

•  Description: 

-  Provides  a  radar  system  operating  in  the  S-band  frequency 
band  to  replace  the  AFSSS  VHF  “Fence”  radar  that  currently 
performs  detection  of  orbiting  space  objects. 

-  The  S-band  radar  will  have  a  modern,  net-centric  architecture 
that  is  capable  of  detecting  much  smaller  objects  in  LEO/MEO. 

-  The  system  will  operate  with  greater  accuracy  and  timeliness 
to  meet  warfighter  requirements  for  SSA. 

-  Two  radar  sites  are  planned,  with  locations  TBD. 

•  Last  Milestone  Awarded:  Milestone  A  (March  14,  2009) 

Sources:  DAES  (as  of  April  IT,  2012)  (FOUO);  Mar  14,  2009  ADM  (FOUO) 
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This  report  documents  the  structured  process  that  was  developed  to  help  project  teams  identify 
acquisition  framing  assumptions  (FAs),  which  are  useful  in  defining  and  tracking  key  program 
assumptions  throughout  the  life  of  an  acquisition  program.  The  process  begins  with  a  workshop, 
in  which  a  briefing  is  used  to  facilitate  a  deliberation  with  the  aim  of  identifying  candidate  FAs. 
The  report  provides  an  introduction  to  the  concept  of  FAs  and  an  overview  of  the  approach 
used  in  the  briefing  to  identify  FAs.  It  includes  an  overview  of  structured  deliberation,  some 
of  the  important  concepts  in  having  a  successful  session,  and  suggestions  on  how  to  tailor 
questions  related  to  program  risk  areas  to  help  elucidate  FAs. 


RAND 


NATIONAL  DEFENSE  RESEARCH  INSTITUTE 


www.rand.org 


ISBN-10  0-8330-8804-1 
ISBN-13  978-0-8330-8804-8 


780833 


088048 


$20.00 

52000 


TL-153-OSD 


9 


